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Abstract 

The analysis of random coding error exponents pertaining to erasure/list decoding, due to 
Forney, is revisited. Instead of using Jensen's inequality as well as some other inequalities in the 
derivation, we demonstrate that an exponentially tight analysis can be carried out by assessing 
the relevant moments of a certain distance enumerator. The resulting bound has the following 
advantages: (i) it is at least as tight as Forney's bound, (ii) under certain symmetry conditions 
associated with the channel and the random coding distribution, it is simpler than Forney's 
bound in the sense that it involves an optimization over one parameter only (rather than two), 
and (iii) in certain special cases, like the binary symmetric channel (BSC), the optimum value 
of this parameter can be found in closed form, and so, there is no need to conduct a numerical 
search. We have not found yet, however, a numerical example where this new bound is strictly 
better than Forney's bound. This may provide an additional evidence to support Forney's 
conjecture that his bound is tight for the average code. We believe that the technique we 
suggest in this paper can be useful in simplifying, and hopefully also improving, exponential 
error bounds in other problem settings as well. 

Index Terms: random coding, erasure, list, error exponent, distance enumerator. 

1 Introduction 



In his celebrated paper [3], Forney extended Gallager's bounding techniques [2] and found exponen- 
tial error bounds for the ensemble performance of optimum generalized decoding rules that include 
the options of erasure, variable size lists, and decision feedback (see also later studies, e.g., [1], 
[4], [5], [6], [8], and [10]). 
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Stated informally, Forney [3] considered a communication system where a code of block length n 
and size M = e"^ {R being the coding rate) , drawn independently at random under a distribution 
{P{x)}, is used for a discrete memoryless channel (DMC) {P{y\x)} and decoded with an erasure/list 
option. For the erasure case, in which we focus hereafter, an optimum tradeoff was sought between 
the probability of erasure (no decoding) and the probability of undetected decoding error. This 
tradeoff is optimally controlled by a threshold parameter T of the function e"^ to which one 
compares the ratio between the likelihood of each hypothesized message and the sum of likelihoods 
of all other messages. If this ratio exceeds e"^ for some message, a decision is made in favor of that 
message, otherwise, an erasure is declared. 

Forney's main result in [3] is a single-letter lower bound Ei{R,T) to the exponent of the 
probability of the event £i of not making the correct decision, namely, either erasing or making the 
wrong decision. This lower bound is given by 



Ei{R, T) = max [Eois, p)-pR- sT] (1) 

0<s<p<l 



where 



Eo{s,p) = - In 



Y,[Y.nx)P'-'{y\x)]- (Y.P{x')P^/^{y\x') 



(2) 



The probability of the undetected error event £2 (i.e., the event of not erasing but making a wrong 
estimate of the transmitted message) is given by E2{R,T) = Ei{R,T) + T.^ As can be seen, the 
computation of Ei{R,T) involves an optimization over two auxiliary parameters, p and s, which 
in general requires a two-dimensional search over these two parameters by some method. This is 

different from Gallager's random coding error exponent function for ordinary decoding (without 



^Forney also provides improved (expurgated) exponents at low rates, but we will focus here solely on (1). 
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erasures), which is given by: 



Er{R)= me^[Eo{p)-pR], (3) 

0<p<l 



with Eo{p) being defined as 



Eo{p) = - In 



y \ X 



(4) 



where there is only one parameter to be optimized. In [3], one of the steps in the derivation involves 
the inequality {J^iO'iY ^ J2iO'h which holds for r < 1 and non-negative {ai} (cf. eq. (90) in [3]), 
and another step (eq. (91e) therein) applies Jensen's inequality. The former inequality introduces 
an additional parameter, denoted p, to be optimized together with the original parameter, s? 

In this paper, we offer a different technique for deriving a lower bound to the exponent of 
the probability of 6i, which avoids the use of these inequalities. Instead, an exponentially tight 
evaluation of the relevant expression is derived by assessing the moments of a certain distance 
enumerator, and so, the resulting bound is at least as tight as Forney's bound. Since the first 
above-mentioned inequality is bypassed, there is no need for the additional parameter p, and so, 
under certain symmetry conditions (that often hold) on the random coding distribution and the 
channel, the resulting bound is not only at least as tight as Forney's bound, but it is also simpler in 
the sense that there is only one parameter to optimize rather than two. Moreover, this optimization 
can be carried out in closed form at least in some special cases like the binary symmetric channel 
(BSC). We have not found yet, however, a convincing^ numerical example where the new bound is 
strictly better than Forney's bound. This may serve as an additional evidence to support Forney's 
conjecture that his bound is tight for the average code. Nevertheless, the question whether there 



^The parameter s is introduced, as in many other derivations, as the power of the likelihood ratio that bounds 
the indicator function of the error event. This point will be elaborated on in Section 4. 

^In a few cases, small differences were found, but these could attributed to insufficient resolution of the two- 
dimensional search for the optimum p and s in Forney's bound. 
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exist situations where the new bound is strictly better, remains open. 

We wish to emphasize that the main message of this contribution, is not merely in the simpli- 
fication of the error exponent bound in this specific problem of decoding with erasures, but more 
importantly, in the analysis technique we offer here, which, we believe, is applicable to quite many 
other problem settings as well. It is conceivable that in some of these problems, the proposed tech- 
nique could not only simply, but perhaps also improve on currently known bounds. The underlying 
ideas behind this technique are inspired from the statistical mechanical point of view on random 
code ensembles, offered in [9] and further elaborated on in [7]. 

The outline of this paper is as follows. In Section 2, we establish notation conventions and give 
some necessary background in more detail. In Section 3, we present the main result along with a 
short discussion. Finally, in Section 4, we provide the detailed derivation of the new bound, first 
for the special case of the BSC, and then more generally. 

2 Notation and Preliminaries 

Throughout this paper, scalar random variables (RV's) will be denoted by capital letters, their 
sample values will be denoted by the respective lower case letters, and their alphabets will be 
denoted by the respective calligraphic letters. A similar convention will apply to random vectors of 
dimension n and their sample values, which will be denoted with same symbols in the bold face font. 
The set of all n-vectors with components taking values in a certain finite alphabet, will be denoted 
as the same alphabet superscripted by n. Thus, for example, a random vector X = {Xi, . . . 
may assume a specific vector value x = G as each component takes values in 

X. Sources and channels will be denoted generically by the letter P or Q. Information theoretic 
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quantities entropies and conditional entropies, will be denoted following the usual conventions of 
the information theory literature, e.g., H(X), H(X\Y), and so on. When we wish to emphasize 
the dependence of the entropy on a certain underlying probability distribution, say Q, we subscript 
it by Q, i.e., use notations like Hq{X), Hq{X\Y), etc. The expectation operator will be denoted 
by E{-}, and once again, when we wish to make the dependence on the underlying distribution 
Q clear, we denote it by Eq{-}. The cardinality of a finite set A will be denoted by |v4|. The 
indicator function of an event 6 will be denoted by !{£}■ For a given sequence y G y being a 
finite alphabet, Py will denote the empirical distribution on y extracted from y, in other words, 
Py is the vector {Py{y), y &y}, where Py{y) is the relative frequency of the letter y in the vector 
y. For two sequences of positive numbers, {a^} and {bn}, the notation = bn means that {a^} 
and {bn} are of the same exponential order, i.e., ^Inf^^Oasn^oo. Similarly, a„ < Oji means 
that limsup„_>oo ^ In |^ < 0, and so on. 

Consider a discrete memoryless channel (DMC) with a finite input alphabet X, finite output 
alphabet y, and single-letter transition probabilities {P{y\x), x e X, ye y}. As the channel is 
fed by an input vector x G X^, it generates an output vector y E y^ according to the sequence 
conditional probability distributions 

P{yi\xi,...,Xi,yi,...,yi-i) = P{yi\xi), i = l,2, ...,n (5) 

where for z = 1, (j/i, . . . , is understood as the null string. A rate-i? block code of length n 
consists of M = e"-^ n-vectors Xm G 'V", m = 1, 2, . . . , M, which represent M different messages. 
We will assume that all possible messages are a-priori equiprobable, i.e., P(m) = 1/M for all 
m = 1,2,...,M. 

A decoder with an erasure option is a partition of y^ into (M + 1) regions, TZq^TZi, . . . ,71m- 
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Such a decoder works as follows: If y falls into 7?.^, m = 1,2,..., M, then a decision is made in 
favor of message number m. If y G TZq, no decision is made and an erasure is declared. We will refer 
to TZq as the erasure event. Given a code C = {xi, . . . , xm} and a decoder TZ = {TZq, TZi, . . . , TZm), 
let us now define two additional undesired events. The event £i is the event of not making the 
right decision. This event is the disjoint union of the erasure event and the event £2, which is the 
undetected error event, namely, the event of making the wrong decision. The probabilities of all 
three events are defined as follows: 

M I M 

= E E ^(^m,y) = ^E E ny\xm) (6) 

m=i yeiz^ m=i ^/g7^^ 

M , M 

Pr{f2} = E E E ^(=«-''?/) = mE E PivM (7) 

m=l yeTZm m'^m m=l yeUm m'^m 

Pr{7^o} = Pr{fi} - Pr{f2}. (8) 

Forney [3] shows, using the Neyman-Pearson theorem, that the best tradeoff between Pr{fi} 
and Pr{£^2} (or, equivalently, between Pr{7^o} and Pr{£^2}) is attained by the decoder TZ* = 
(n*o,ni, . . . ,n*M) defined by 

= \y: ^ , , . > e \, m=l,2,...,M 

M 

K = n(^mr, (9) 

m=l 

where (T^-j^)'^ is the complement of 7?.^, and where T > is a parameter, henceforth referred to as 
the threshold, which controls the balance between the probabilities of £1 and £2- Forney devotes the 
remaining part of his paper [3] to derive lower bounds, as well as to investigate properties, of the 
random coding exponents (associated with 7^*), Ei{R, T) and E2{R, T), of Pr{£^i} and Pr{£^2}, the 
average probabilities of £1 and £2, respectively, (w.r.t.) the ensemble of randomly selected codes, 
drawn independently according to an i.i.d. distribution P{x) = YVi^i P{xi). As mentioned in the 
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Introduction, Ei{R,T) is given by (1) and E2{R,T) = Ei{R,T)+T. 

3 Main Result 

Our main result in this paper is the following: 

Theorem 1 Assume that the random coding distribution {P{x), x € X} and the channel transition 
matrix {P{y\x), x E X, y & y} are such that for every real s, 



lyis) = - In 



X 



(10) 



is independent of y, in which case, it will be denoted by 7(5). Let sr be the solution to the equation 



7(s) - sj'is) = R, 



(11) 



where 7'(s) is the derivative 0/7(5). Finally, let 



El{R, T, s) = A{R, s) + 7(1 -s)-sT- In \y\ 



(12) 



where 



Then, 



A{R,s) 



7(5) - R s> SR 
sj'{sr) s < sr 



(13) 



(14) 



where El{R,T) = sup^>qEI{R,T,s) and 



(15) 



where E^{R,T) = El{R,T)+T. Also, El{R,T) > Ei{R,T), where Ei{R,T) is given in (1). 



Three comments are in order regarding the condition that 7y(s) of eq. (10) is independent of y. 

The first comment is that this condition is obviously satisfied when {P{x)} is uniform and the 
columns of the matrix {a^y} = {P{y\x)} are permutations of each other, because then the sum- 
mations X^j. P{x)P^{y\x), for the various j/'s, consist of exactly the same terms, just in a different 
order. This is the case, for example, when X = y iso. group endowed with an addition/subtraction 
operation (e.g., addition/subtraction modulo the alphabet size), and the channel is additive in 
the sense that the 'noise' iY — X) is statistically independent of X. Somewhat more generally, 
the condition 7j/(s) = 7(5) for all y holds when the different columns of the matrix {P(y|x)} are 
formed by permutations of each other subject to the following rule: P{y\x) can be permuted with 
P{y\x') if P{x) = P{x'). For example, let X = {A,B,C} and y = {1,2}, let the random coding 
distribution be given by P{A) = a, P{B) = P{C) = (1 — a)/2, and let the channel be given by 
P{Q\A) = P{l\A) = 1/2, P(0|5) = P{l\C) = 1 - P(0|C) = 1 - P{l\B) = h. In this case, the 
condition is satisfied and 7(5) = - ln[(l - a)/2){¥ + (1 - bf ) + a ■ 2"*]. 

The second comment is that the derivation of the bound, using the proposed technique, can be 
carried out, in principle, even without this condition on 7y(s). In the absence of this condition, 
one ends up with an exponential expression that depends, for each y, on the empirical distribution 
Py, and its summation over y can then be handled using the method of types, which involves 
optimization over {Py}, or in the limit of large n, optimization over the continuum of probability 
distributions on y. But then we are loosing the simplicity of the bound relative to Forney's bound, 
since this optimization is at least as complicated as the optimization over the additional parameter 
p in [3]. 
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Our last comment, in the context of this condition on 7j/(s), is that even when it holds, it is 
not apparent that the expression of Forney's bound Ei [R, T) can be simplified directly in a trivial 
manner, nor can we see how the optimum parameters p and s can be found analytically in closed 
form. This is true even in the simplest case of the BSC. 

4 Derivation of the New Bound 
4.1 Background 

The first few steps of the derivation are similar to those in [3]: For a given code and for every s > 0, 

1 M 

M^i} = mE E nv\xm) 

nsT M ( \ 

' E E p'-\yM E nvM . (16) 



M 

m=l y&y-^ \m'^m 



As for £2, we have similarly. 



Pr:{£2} < e-"(i-^)^ ^ P'-%y\X^) ^ P{y\X^') • (17) 

Since this differs from the bound on Prj^"!} only by the constant factor e""-^, it will be sufficient 
to focus on £1 only. Taking now the expectation w.r.t. the ensemble of codes, and using the fact 
that Xjn is independent of all other codewords, we get: 

M^i} < e-^ Yl E{P'-%y\Xm)} • ^ ( f E ny\Xm')] ] • (18) 
The first factor of the summand is easy to handle: 

E{P^-%y\X^)} = Yl P{x)P'-^{y\x) 
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n 



i=i xex 



— g-W7(l-s)_ 



(19) 



Concerning the second factor of the summand, Forney's approach is to use the inequahty {Y,i o-iY < 
J^iO-h which holds when {a^} are positive and r < 1, in order to upper bound 



and then (similarly to Gallager) use Jensen's inequality to insert the expectation into the bracketed 
expression, which is allowed by limiting p to be less than unity. In other words, the above expression 
is further upper bounded in [3] by 



The idea here, as we understand it, is that the parameter p controls the tradeoff between the gap 
pertaining to the first inequality and the one associated with the second inequality. The first gap 
is maximum when p = 1 and non-existent when p = s {s < 1), whereas for the second gap it is 
vice versa. 

We, on the other hand, will use a different route, where all steps of the derivation will be 
clearly exponentially tight, and without introducing the additional parameter p. To simplify the 
exposition and make it easier to gain some geometrical insight, it will be instructive to begin with 
the special case of the BSC and the uniform random coding distribution. The extension to more 
general DMC's and random coding distributions will be given in Subsection 4.3. Readers who are 
interested in the more general case only may skip to Subsection 4.3 without loss of continuity. 




by 
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4.2 The BSC with the uniform random coding distribution 

Consider the special case where X = y = {0,1}, the channel is a BSC with a crossover probability p, 
and the random coding distribution is uniform over the Hamming space {0, 1}", namely, P{x) = 2~"^ 
for all X G {0, l}". First, concerning the first factor in the summand of (18), we have, in this special 



case: 

r 1 1 1 



7(1 — s) = — In 



l/-+l(l_p)i- 



ln2-ln[p^-* + (l-_p)i-^]. (20) 



As for the second factor, we proceed as follows. Define /? = In and for a given y, let Ny{d) 
denote distance enumerator relative to y, that is, the number of incorrect codewords {x^', fn' / m} 
at Hamming distance d from y. We then have: 



e\ ny\Xm')\ = E 



d=0 

n 



= (l-p)-5^£;{iV^(d)}e-^^<^. (21) 

d=0 

The second (exponential) equality is the first main point in our approach: It holds, even before 
taking the expectations, because the summation over d consists of a subexponential number of 
terms, and so, both [J2d^yid)^~'^'^Y J2d^yid)s~^^'^ are of the same exponential order as 
msi:x.^ Ny{d)e~^^'^ = [majCd ^y{d)e~^'^Y . This is different from the original summation over m', 
which contains an exponential number of terms. Thus, the key issue here is how to assess the 
power-s moments of the distance enumerator Ny{d). To this end, we have to distinguish between 
two ranges of d, or equivalently, 5 = d/n. Let 5gv{R) denote the normalized Gilbert- Varshamov 
(GV) distance, Sgv = dav /'n, i.e., the smaller solution, 5, to the equation 

h{5) =ln2-i2. 
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where 

h{S) = -5ln5- {l-6)ln{l-6), Se [0,1]. 

Now, the second main point of the proposed analysis approach is that E{Ny(d)} behaves 
differently^ for the case Sgv{1^) < <^ < 1 — Sgv{R) ^'^d for the case S < (5(3y(i?) or 5 > 1 — (5(3y(i?). 
Let us define then Gr = {5 : 5gv{R) < < 1 — 5gv{R)}- In particular, using the large deviations 
behavior of Ny{n5), S G [0,1], as the sum of e"^ — 1 binary i.i.d. RV's, it is easily seen (see 
Appendix) that 

r ns[R+h{S)~hi2] X g Co 

EimnS)} = I l^[n^,^s)-in2] } I gl (22) 

Thus, 



^ns[R+h{S)-ln2] _ g-/3sn<5 _|_ ^ ^n[R+h{d) -In 2] _ ^-^snS 

^ns{R-\n2) . exp{ns max[/i(5) - (55]} + e'^^-^-^'^^) . exp{nmax[/i(5) - I3s6]} 



(23) 



We are assuming, of course, < C = ln2 — h{p), which is equivalent to p < 5gv{R) ot p > 
1 — 5gv{R)- We also assume, for the sake of simplicity and without essential loss of generality, 
that p < 1/2, which will leave us only with the first possibility of p < 5gv{R)- Therefore, the 
global (unconstrained) maximum of h{S) — pS, which is attained at 6 = p, falls outside Qr, and so, 
max5gg^[/i(5) — P5] is attained at 6 = 6gv{R) which yields 



max[h{6) - pS] = h{5Gv{R)) - P^GviR) = ln2-R- I35gv{R)- 



^The intuition behind this different behavior is that when h{S) + i? — ln2 > 0, the RV Ny{d), which is the sum 

of e"^ — 1 many i.i.d. binary RV's, l{d{Xm' , y) ~ d}, concentrates extremely (doublc-cxponentially) rapidly around 
its expectation Qn[R+h(5)-\n2] ^ whereas for h{5) + i? — ln2 < 0, Ny(d) is typically zero, and so, the dominant term of 
E{Nlj{d)} is r • Pr{iVy(d) = 1} « e"[«+''(''>-'"21. 
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Thus, the first term in the large square brackets of the r.h.s. of (23) is of the exponential order 

of e~"*^'^G'v(^)_ As for the second term, the unconstrained maximum of h{6) — /3sS is obtained at 
S = Ps = ps^^i_p)s , which can be either larger or smaller than 5gv{R)-, depending on s. Specifically, 

max[/^(5) - I3s8\ = | f ^^^^ '/'P^ , , ^ ^^^S^| (24) 

The condition ps < Sgv{R) is equivalent to 

A ln[{l -Sgv{R))/6gv{R)] 

S>Sr = — ^-^^ 

Thus, the second term in the square brackets of the r.h.s. of eq. (23) is of the order of ^-'^K^'^) ^ 
where 

M^,i?) = l '^^^'^.U '-'^ (25) 
' ^ [ I3s5gv{R) s<sr ^ ^ 

and where 

IJ:o{s,R) = Psps - h{ps) + ln2 - R 

= sln(l-p)-ln[p" + (l-p)^]+ln2-ii:. (26) 

Next, observe that the second term, e"""'^^*'^), is always the dominant term: For s < sr, this is 
trivial as both terms behave like g-^^^^aviR) _ ^or s > sr (namely, ps < dGviR)), bs S = ps achieves 
the global minimum of the function f{5) = (3s6 — h{S) + In 2 — i?, we have 

fiois,R) = f{ps) < f{dGv{R)) = psdGviR). 

Therefore, we have established that 

^Ujl Pivl^m')^ I =exp|-n s\nj^ + fi{s,R) | (27) 
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independently of y. Finally, we get: 

I ^ ^-nE^(R,T,s) (28) 

where 

El {R, T, s) = i?) + s In - \n\p^-' + (1 - p)^''] - sT. 

It is immediate to verify that this coincides with the expression of Theorem 1 when specialized to 
the case of the BSC with a uniform random coding distribution. 

We next derive closed form expressions for the optimum value of s, denoted SQp^, using the 
following consideration: We have seen that Ef{R,T, s) is given by 

F{s) = Ms, R) + sln ln[pi-^ + (1 - p^-'] - sT 

1-p 

for s > SR, and by 

G{s) = f3s5Gv{R) + s In - \u\p^-' + (1 - pf''] - sT 

for s < SR. Both F{s) and G{s) are easily seen to be concave functions and hence have a unique 
maximum each, which can be found by equating the corresponding derivative to zero. We have 
also seen that F{s) < G{s) for all s, with equality at s = sr and only at that point. This means 
that F{s) and G{s) are tangential to each other at s = sr, in other words, F{sr) = G{sr) and 
P'{sr) = G'{sr), where F' and G' are the derivatives of F and G, respectively. Now, there are three 
possibilities: If F'{sr) = G'{sr) = 0, then Sq^^ = sr. If F'{sr) = G'{sr) < 0, then Sgpt < 
is found by solving the equation G'{s) = 0. If F'{sr) = G'{sr) > 0, then s^pi > sr is found by 
solving the equation F'{s) = 0. 

Let us assume first that Sgp^ < sr. Then, the equation G'{s) = is equivalent to: 
(3dGv{R) + In + Pi-s Inp + (1 - pi-s) ln(l - p) - T = 

14 



-n 



s In ■ 



1-p 



+ n{s,R) 



or 



whose solution is: 



(3 (35gv{R)-T ■ 
Of course, if the r.h.s. of (29) turns out to be negative, then Sgpt = 0. Thus, overah 



«opt = si{p,R,T) 



1 P{l-dav{R))+T 
(3 P5gv{R)-T , 



+ 



where = maxja;, 0}. Of course, when Sq^^ = 0, the new bound El{R, T) vanishes. 
Next, assume that Sgp^ > sr. In this case, 

Ei{R,T,s) = F{s) 

= In 2 - ln[p" + (1 - pY] - ln\p^-' + (1 - p)^-'] -R-sT. 



Thus, the optimum s minimizes the convex function 



f{s) = \n[p' + {l-py]+ln\p'-' + {l-py-']+sT 



= In 



1 + il-p) 



p 



1-p 



+ p 



1 — p 
p 



+ sT. 



Equating the derivative to zero, we get: 



or equivalently, defining z = e^^ as the unknown, we get: 

— {1 — p)/z + pz T 



l + {l-p)/z+pz /?' 
which is a quadratic equation whose relevant (positive) solution is: 



Z = Zq 



2p{T + (5) 
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provided^ that T < (3, and so the derivative vanishes at 



5opt = S2{p,T) == 



It is not difficult to verify that Sq^^ never exceeds unity. Also, SQp^ is always positive {zq > 1) 
since the condition F'{sr) > 0, which is equivalent to the condition T < (3{psii ~ P'^-sr), implies 
T < (3/2, which in turn is the condition for Sgp^ > 0. Note that for T = 0, we obtain S2{p, 0) = 1/2, 
in agreement with the Bhattacharyya bound. 

In summary, the behavior of the solution can be described as follows: As R increases from to 
C = ln2 — h{p), sr increases correspondingly from to 1, and so, the expression P{Psr ~ Pi-sr) 
(which is positive as long as i? < In 2 — /i(pi/2)) decreases. As long as this expression is still larger 
than T, we have F'{sr) > and the relevant expression of El(R, T, s) is F(s), which is maximized 
at s = S2{p, T) independently of R. At this range, the slope of E*{R, T), as a function of i?, is —1. 
As R continues to increase, we cross the point where F'{sr) = (a point which can be thought of 
as an analogue to the critical rate of ordinary decoding) and enter into the region where F'{sr) < 0, 
for which El{R,T) = G{si{p, R,T)). 

4.3 More General DMC's and Random Coding Distributions 

Assume now a general DMC {P{y\x), x & X, y & y} and a general i.i.d. random coding distribution 
P{x) = Ili=iP{xi) that satisfy the condition of Theorem 1. As for the second factor of the 
summand of (18), we have the following: 

eI(j2 PivlXm')] I = El(j2^y(Q4y)-^MnEQlnPiY\X)} 

[ \m'j^m J ) [ VQx|!/ / 

= J2E{N^{Q,^y)}-eMnsEQ\nP{Y\X)}, (34) 

Qx\y 



^Note that if T > p, the decoder will always erase (even for R = 0) since for p < 1/2, we have 



P{y\xm)/[j:^,,^P{y\x^')] < (1 -p)7p" = e^" < e-^ 
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where Ny{Q^^y) is the number of incorrect codewords whose conditional empirical distribution with 

y is Qx\y and Eq is the expectation operator associated with Py x Qx\y Define 

Qr = {Qxiy : R + Hq{X\Y) + Eq InP(x) > 0}, 

where Hq{X\Y) is the conditional entropy induced by Py x Qx\y Analogously to the case of the 
BSC (see also Appendix), we have: 



rmun ^^ ^ i ^Mns[R + Hq{X\Y) + Eq In P{x)]} Qx\y Gr 
J^ii^y{i^xly)i I ^^p{ri[R + HQ{X\Y) + EQlnP{x)]} Qx\y^Q'k 



(35) 



Thus, 



e\\Y^ P{y\X^,)\ \ = eMns[R + HQ{X\Y) + EQlnP{x)]}x 

e^v{nsEQ lnP(y|X)} + 

^ c^^{n[R + HQ{X\Y)+EQhiP{x)]}x 
exp{nsEQ lnP(y|X)} 
= A + B. (36) 



As for A, we obtain: 

A = exp{ns[R+ max {Hq{X\Y) + EQln[P{X)P{Y\X)])]} (37) 

Note that without the constraint Qx\y G Gr, the maximum of {Hq(X\Y) + EQln[P{X)P{Y\X)]) 
is attained at 

^ /IN / I X A P(x)P(y\x) 

Qxiy{x\y) = Pxiy{x\y) = ^^^^ p(,.)p(^|,.) • 

But since R < I{X;Y), then P^^y is in Q'^. We argue then that the optimum Qx\y in Gr is on the 
boundary of Gr, i-e., it satisfies R + iJQ(X|y) + Eq In P(X) = 0. To see why this is true, consider 
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the following argument: Let Q^i , be any internal point in Qji and consider the conditional pmf 

x\y 

Q* = (1 - t)Ql^y + tP^\y, t G [0, 1]. Define f{t) = HQt{X\Y) + Egt \iv[P{X)P{Y\X)]. Obviously, 
/ is concave and /(O) < /(I). Now, since (5° G Qr and = Py.\y G then by the continuity 
of the function R + HQt{X\Y) + Eqt InP(X), there must be some t = to for which is on the 
boundary of Qr. By the concavity of /, f{to) > (1 — io)/(0) + io/(l) > /(O)- Thus, any internal 
point of Gr can be improved by a point on the boundary between Qr and Qj^. Therefore, we have 



m^ {HQ{X\Y) + EQln[P{X)P{Y\Xm 

max [HQiX\Y) + EQlnPiX) + EQ\nP{Y\X)] 

ifQ(X|y)+£;QlnP(X)=-R} 

max [-R + Eq In P(y |X)] 

{Q.\y-- HQ{X\Y)+EQlnP{X)=-R} 

= -R+ max EqIiiP{Y\X) 

{Q,|^: HQ(X|y)+i;plnP(X)=-ii} 

= max EqlnPiYlX) (38) 



which means that A = e ^^^(^) ^ where 



A(i?)= min EQln[l/P{Y\X)]. 



The achiever of A(i?) is of the form 

P{x)P^'^{y\x) 



QiAy) 



where sr is such that Hq{X\Y) + EqlnP^X) = —R, or equivalently, sr is the solution ^ to the 
equation s'y'{s) — 7(5) = R. In other words. 



"Observe that for s = 0, Hq{X\Y) + EqIu P{X) = and for s = 1, Hq{X\Y) + Egln P{X) = -I{X;Y) < -R. 
Thus for R < I{X;Y), sr £ [0, 1). 
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Considering next the expression of we have: 

B = exp{n[R + max^ {Hq{X\Y) + Eq InP(X) + sEq lnP(Y\X))]}. 

Qx\y&S'ji 

The unconstrained maximizer of {Hq{X\Y) + Eq lnP{X) + sEq lnP{Y\X)) is 

(s) 

Now, there are two cases, depending on the value of s: If s is such that Q^-^y £ Qr, or equivalently, 
s > sr, then B = e~"'['''(*)~^l. If Q^^^ G Gr, namely, s < sr, then once again, the optimum is 
attained at the boundary between Qr and Qj^, and then B = e~'^^'^'^^^\ In summary, B = q-^HR^^)^ 
where 

A{R,s) = 



7(s) - R s> SR 
s7'{sr) s<sr 



The dominant term between A and B is obviously always B because it is either of the same 
exponential order of A, in the case s < sr, or has a slower exponential decay, when s > sr, as then 
the global (unconstrained) maximum of [Hq{X\Y) + EQlnP{X) + sEQlnP{Y\X)] is achieved. 
Thus, putting it all together, we get: 

= e-'*^i*(^'^'^) (39) 
and the optimum s > gives {R, T) . 
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Appendix 



We begin with a simple large deviations bound regarding the distance enumerator. In fact, this 
bound (in a slightly different form) was given already in [7], but we present here too for the sake 
of completeness. For o, 6 G [0, 1], consider the binary divergence 



D{a\\b) = aln^ + (l-a)ln^ 

1 — 



= a In — + (1 — a) In 



b — a 

1 + 



1-6 

To derive a lower bound to D{a\\b), let us use the inequality 



(A.l) 



and then 



D{a\\b) > aln- + (l-a) ^ ^'^ ' 



b ' ' l + (6-a)/(l-6) 

= ain- + o — a 

b 

> a(ln^-iy (A.3) 

Consider first the binary case (the extension to the general case is straightforward as will be 
explained below). For every given y, Ny[d) is the sum of the e**^ — 1 independent binary random 
variables, {l{d{X^',y) = d}}m'^rm where the probability that d{X^',y) = nS is exponentially 
b = e-"[''^2-ft(5)]_ rp^g g^g^^ Ny{n6) > e"^, for A G [0,R), means that the relative frequency of the 
event l{d{X^r,y) = n5} is at least a = e~"(^~"^). Thus, by the Chernoff bound: 

PT{Ny{n6) > e"^} < exp {-(e"^ - l)L>(e-"(-^-^) ||g-n[in2-ft(5)])| 

< exp I -e"^ • e-"(^-^) (n[(ln 2-R- h{S) + A] - 1)} 

< exp{-e"^(n[ln2-i?- /i(5) + A] - 1)} . (A.4) 
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Therefore, for S G we have: 

E{N^{nS)} < e"" • Pr{l < Ny{nS) < e"^} + e""^ ■ Fv{Ny{nS) > e^'} 

< e'^" • Pi{Ny{n6) > 1} + e""^' ■ Pi{Ny{n5) > e"^} 

< e"" • E{Nyin6)} + e""^ ■ e-^"^-^)^"' 

< g"" . g"[-R+M'5)-ln2] _|_ gnRs . g-(ne-l)e"«_ (A.5) 

One can let e vanish with n sufficiently slowly that the second term is still superexponentially 
small, e.g., e = l/^n. Thus, for 6 G g%, E{N^{n6)} is exponentially bounded by e"[^+'*W-i°2] 
independently of s. For (5 G ^ij, we have: 

£;{Ar^(n(5)} < e"^[^+'^('')-^''2+^l • Pr{iVy(n(5) < e"[-^+'*(^)-i°2+6]| ^ 

gnfc . Pr{Ar^(„(5) > gn[ii+/x(5)-ln2+e]| 
< gns[il+?i(<5)-ln2+e] _|_ ^nRs . g-(ne-l)e"^ (^^_q^ 

where again, the second term is exponentially negligible. 

To see that both bounds are exponentially tight, consider the following lower bounds. For 

E{Nlj{n6)} > I' ■ Fi{Ny{n6) = 1} 

= • PT{dH{X,y) = nS} ■ [1 - Pr{d^(X,y) = nSjY""-^ 

_ gnRg-n[ln2-/i(5)] _ ^ _ g-n[ln2-/i(5)]l 

= e"[«+'^('^)-i°2] • expje'^^lnil - e-^P^^-M^)]]}. (a.7) 
Using again the inequality in (A. 2), the second factor is lower bounded by 

exp{-e"^e-"[''^2-'^(^)V(l " e'^l^'^^-'^^^)!)} = exp{-e-"P°2-^-'^('^)l/(l " e-"!'"^^-'^^'')])} 
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which clearly tends to unity as In 2 — i? — h{S) > for 5 G Gj^. Thus, E{Ny{nS)} is exponentially 
lower bounded by e"[^+'»('5)-in2]_ Yqt S G Qr, and an arbitrarily small e > 0, we have: 

^ ^nslR+h{S)-ln2-e] . f^^ _ p^^Ny{nd) < g^I^+M^)-!" 2-.] |^ (A.8) 

where Pr{%(n5) < e"[^+'*('^)-i°2-e]} 

is again upper bounded, for an internal point in Qn, by 
a double exponentially small quantity as above. For S near the boundary of Qr, namely, when 
R + h{6) — In 2 PS 0, we can lower bound E{Ny(n6)} by slightly reducing R to R' = R — e (where 
e > is very small). This will make 6 an internal point of for which the previous bound applies, 
and this bound is of the exponential order of ^^IR'+K^)-^^'^] . Since R' + h{5) — In 2 is still very close 
to zero, then e,'"'[R'+H^)-^^'^] [g of the same exponential order as e.'^4^+H^)-^^'^] since both are about 

The above proof extends straightforwardly from the binary case to the more general case. 
The only difference is that in the general case, for a given y, the probability that a random 
codeword, drawn under {P(x)}, would have a given conditional empirical distribution Q^iy with y, 
is exponentially e"l^<5('''^l^)+-^Q inP(x)] ^hus, h{5) — In 2 of the binary case has to be replaced by 
Hq{X\Y) + Eq lnP{X) in all places. 
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